ON  A  ROOTING  PROBLEM 
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Given  a  set  of  N  cities,  with  every  two  linked  by  a  road, 
and  the  tines  required  to  traverse  these  roads,  we  wish  to 
determine  the  path  fron  one  given  city  to  another  given  city 
which  minimizes  the  travel  time.  The  times  are  not  directly 
proportional  to  the  distances  due  to  varying  quality  of  roads, 
and  varying  quantities  of  traffic. 

The  functional  equation  technique  of  dynamic  programming, 
combined  with  approximation  In  policy  space,  yield  an  Iterative 
algorithm  which  converges  after  at  most  (N — 1 )  Iterations. 
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ON  A  ROUTING  PROBLEM 
By 

Richard  Bellman 


§1 ,  Introduction, 

The  problem  we  wish  to  treat  is  a  combinatorial  one  involving 
the  determination  of  an  optimal  route  from  one  point  to  another. 
These  problems  are  usually  difficult  when  we  allow  a  continuum, 
and  when  we  admit  only  a  discrete  set  of  paths,  as  we  shall  do 
below,  they  are  notoriously  so. 

The  purpose  of  this  paper  is  to  show  that  the  functional 
equation  technique  of  dynamic  programming,  [l] , [2] ,  combined 
with  the  concept  of  approximation  In  policy  space,  yields  a  method 
of  successive  approximations  which  is  readily  accessible  to  either 
hand  or  machine  computation  for  problems  of  realistic  magnitude. 

The  method  ie  distinguished  by  the  fact  that  it  Is  a  method  of 
exhaustion,  l.e.  it  converges  after  a  finite  number  of  iterations, 
bounded  in  advance. 

§2.  Formulation . 

Consider  a  set  of  N  cities,  numbered  in  some  arbitrary  fashion 
from  1  to  N,  with  every  two  linked  oy  a  airect  road.  Tne  time 
required  to  travel  from  1  to  J  is  not  directly  proportional  to 
the  distance  between  1  and  due  to  road  conditions  and  traffic. 
Given  the  matrix  T  »  ( t ^  5 ) ,  not  necessarily  symmetric,  wnere  t^ 

Is  the  time  required  to  travel  from  i  to  j,  we  wish  to  trace  a 
path  between  1  and  N  wnich  consumes  minimum  time. 
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Since  there  are  only  a  finite  number  of  paths  available, 
the  problem  reduces  to  choosing  the  smallest  from  a  finite  set 
of  numbers.  This  direct,  or  enumeratlve,  approach  is  impossible 
to  execute,  however,  for  values  of  N  of  the  order  of  magnitude 
of  20. 

We  shall  construct  a  search  technique  which  greatly  reduces 
the  time  required  to  find  minimal  paths. 


§j5.  Punctlonal  Equation  Approach. 

Let  us  now  Introduce  a  dynamic  programming  approach.  Let 


(1)  f^  ■  the  time  required  to  travel  from  i  to  N,  i-1,2, . . . ,N— 1, 
using  an  optimal  policy, 


with  fN  ■  0. 


Employing  the  principle  of  optimality, 
satisfy  the  nonlinear  system  of  equations 


(2) 


Min 

>1 


L  fciJ  +  fjJ’ 


1*1 » 2 , • • 


we  see  that  the  f^ 

,N-1» 


m  0. 


This  system  differs  from  the  usual  syetems  encountered  in 
iynam4  ^  programming  in  that  we  do  not  nave  a  ready  computational 
scheme . 


§4.  Uniqueness . 

Let  us  show  that  there  exists  at  most  one  solution  of  the 
system  in  (J.2). 

Assume  that  [f^j  and  ^  |  are  two  solutions,  with  «  0, 

and  let  k  be  an  index  for  which  f^-F^  achieves  it  maximum.  Then 

(i)  r*-  + 


Min 

J4k 
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Let  the  minimum  In  the  first  equation  be  assumed  for  .1  *  r,  and 
the  second  fbrj  *  s.  It  Is  clear,  since  t^j  >  0  for  all  1,4, 
that  r  t  k,  s  f  k.  Then  we  have  the  equalities  and  Inequalities: 


fk  *  ‘kr  +  fr  ^  ‘its  +  V 


pk  *  * 

These  lead  to 


k  “  ''ka  *  ?b  ^  vkr 


t,„  +  F  . 
r 


(3) 


2 


Since  k  was  an  index  where  f^  -  F^  achieved  its  maximum,  we  must 
have 

w  fk-Flt-f.  *V 

which  can  only  be  true  If  In  (2)  we  have 
(5)  tk  -  tM  ♦  f. 

Now  repeat  this  procedure  for  the  pair  .  It 

follows,  from  the  foregoing  argument,  that  there  must  be  another 

pair  ff  ,P  J  with  f  -  P  ■  f  -  Pm  -  f.  -  P,  .  Furthermore, 
l  p  p  J  ppsskk 

p  f  s,  and  p  f  k,  since  we  have 

<6>  fk  -  ‘k.  +  ‘.P  +  v 

Preceding  In  this  way,  we  exhaust  the  set  of  points  1*1, 
2,...,N-1,  with  the  result  that  one  of  the  terms  lr  the  continued 
equality  above  must  be  f^  —  PN  -  0.  Hence  f^  *  F^  for  1-1 ,2, . . . ,N-1 . 
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§3*  Approximation  in  Policy  Space . 

Let  us  now  turn  to  the  problem  of  determining  an  algorithm 
for  obtaining  the  solution  of  the  system  In  (3.2),  The  baelc 
method  is  that  of  successive  approximations.  We  choose  an  initial 
sequence  {f^J  *  and  then  proceed  iteratively,  setting 


(1) 


f (k4l) 
*1 


f(k+l) 


Min  f  +  f^k)  I,  1-1, M-l, 

0, 


for  k-0, 1,2, . . . ,  . 

The  choice  of  J  seems  to  require  some  care.  Let  us 

then  invoke  the  concept  of  approximation  In  policy  space  lr.  order 
to  obtain  a  sequence  which  Is  monotone  Increasing. 

Perhaps  the  simplest  policy  that  we  can  employ  Is  to  proceed 
directly  from  i  to  N.  Define 


(2)  f^°)  -  t1N,  1-1,2, ...,N. 


It  follows  that  fp^  as  defined  by 


(3)  fp>.  Min  [t1J  +  fj0)],  1-1.2... 


.  ,  N  1 , 


J41 

*N  '  °' 

satisfies  the  irequallty 


(4)  fp*  <  1-1,2, 

This  Inequality  is  iaraedlate  alien  we  realize  that  fp/  re pre— 
sents  the  minimum  time  for  a  path  with  at  most  one  stop.  It 

r  'u )  1 

follows  then  Inductively  that  the  sequences  |f^  '  j  as  defined  by 
(1),  with  fj0^  as  in  (2),  satisfy  the  Inequalities 
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(5)  <  fjk\  1-1,2, . . .  ,N,  k-0,1,2, .  .  • 

It  Is  important  to  note  that  there  are  many  other  policiee 

# 

we  could  employ  to  obtain  monotone  convergence. 

It  follows  that 

(6)  11m  f.^  -  f.,  1-1, 2,... ,N, 

k— >cd  1  1 

furnishing  a  solution  to  (3*2). 

It  is  clear  from  the  physical  interpretation  of  this  itera¬ 
tive  scheme  that  at  most  (N-l)  iterations  are  required  for  the 
sequence  to  converge  to  the  solution. 

§6.  Computational  Aspects . 

It  1»  easily  seen  that  the  iterative  scheme  discussed  above 
is  a  feasible  method  for  either  hand  or  machine  computation  for 
values  of  N  of  the  order  of  magnitude  of  30  or  100. 

For  each  1,  we  require  only  the  column  (t4  4),  J-1,2,...,N 
of  the  matrix  T.  Hence,  the  memory  requirement  for  a  digital 
computer  is  small. 

§7.  Monotone  Increasing  Convergence . 

Turning  bacic  to  (3  •2),  let  us  consider  the  sequence  of 
approximations  defined  by 

(1)  f{°^  -  Min  t14,  i-1,2, . . . ,N-1, 

>i 

4°}  -  °» 

ff**^  -  Min  '  t,  4  +  flk)  |,  1-1,2, ...  ,N— 1, 
l  Jfi  l  1-!  J  J 

fik+1^  -  0. 


•  I  owe  this  choice  of  an  initial  policy  to  F.  Haight. 
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It  is  clear  that  It  la  not,  however, 

obvious  that  this  method  yields  a  uniformly  bounded  sequence. 
To  establish  tnla,  let  us  show,  Inductively,  that 

(2)  1  1*1,2, . . . ,N,  k-0,1,2,..., 


where  /f ^ J  is  tne  aolution  of  (3.2). 


Tne  Inequality  Is  certainly  true  for  k  «  0.  Hence,  assuming 
that  it  holds  for  k,  we  have 


(3)  -  Min 

J*1 

<  fi* 


*  f00  1  < 
rJ  J  * 


Min 

J*i 


It  follows  that  the  sequence  £f  j1*'  J  converges  to  |fj  J 
as  k  ->  ao  ,  furnishing  the  desired  monotone  convergence.  Once 
again,  only  a  finite  number  of  Iterations  will  be  required.  It 
Is  to  be  expected  that  the  first  method  will  converge  more 
rapidly . 
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